use:
mex -compatibleArrayDims rcmgmex.c rcmg.c

to main compability with 32 bit, see mex -setup for details.